Recently, FuseLLM introduced the concept of knowledge fusion to transfer the collective knowledge of multiple structurally varied LLMs into a target LLM through lightweight continual training. In this report, we extend the scalability and flexibility of the FuseLLM framework to realize the fusion of chat LLMs, resulting in FusionChat. FusionChat comprises two main stages. Firstly, we undertake knowledge fusion for structurally and scale-varied source LLMs to derive multiple target LLMs of identical structure and size via lightweight fine-tuning. Then, these target LLMs are merged within the parameter space, wherein we propose a novel method for determining the merging weights based on the variation ratio of parameter matrices before and after fine-tuning. We validate our approach using three prominent chat LLMs with diverse architectures and scales, namely NH2-Mixtral-8x7B, NH2-Solar-10.7B, and OpenChat-3.5-7B. Experimental results spanning various chat domains demonstrate the superiority of FusionChat-7B across a broad spectrum of chat LLMs at 7B and 34B scales, even surpassing GPT-3.5 (March) and approaching Mixtral-8x7B-Instruct.

Instead, do the work to present statistical conclusions with uncertainty rather than as dichotomies. Also, remember that most effects can't be zero (at least in social science and public health), and that an "effect" is usually a mean in a population (or something similar such as a regression coefficient)--a fact that seems to be lost from consciousness when researchers slip into binary statements about there being "an effect" or "no effect" as if they are writing about constants of nature. Again, it will be difficult to resolve the many problems with p-values and "statistical significance" without addressing the mistaken goal of certainty which such methods have been used to pursue.